Stereo auto-calibration from structure-from-motion

ABSTRACT

Auto-calibration of stereo cameras installable behind the windshield of a host vehicle and oriented to view the environment through the windshield. Multiple first image points are located of one of the first images captured from the first camera at a first time and matched with first image points of at least one other of the first images captured from the first camera at a second time to produce pairs of corresponding first image points respectively in the first images captured at the different times. World coordinates are computed from the corresponding first image points. Second image points in the second images captured from the second camera are matched to at least a portion of the first image points. The world coordinates as determined from the first camera are used, to solve for camera parameters of the second camera from the matching second image points of the second camera.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority from U.S. provisional patentapplication 61/908,831 filed on 26 Nov. 2013 by the same inventors, thedisclosure of which is included herein by reference.

BACKGROUND

1. Technical Field

The present disclosure relates to a method for calibration stereocameras and in particular for use inside a vehicle as part of a driverassistance system.

2. Description of Related Art

Stereo vision is the process of recovering depth from camera images bycomparing two or more views of the same scene. Binocular stereo uses twoimages, taken with cameras that are separated by a horizontal distanceknown as the “baseline”. Calibrating the stereo camera system allowscomputation of three-dimensional world points in actual units, e.g.millimeters, relative to the cameras based on the image coordinates.

Calibration of a stereo camera system involves the estimation ofextrinsic parameters which describe translation and rotation of thesecond camera relative to the first camera and intrinsic parameters ofeach camera. Intrinsic parameters include focal lengths, principalpoints and other parameters which describe camera image distortion.Image distortion means that image points are displaced from the positionpredicted by an ideal pinhole projection model. The most common form ofdistortion is radial distortion, which is inherent in all single-elementlenses. Under radial distortion, e.g. pincushion distortion and/orbarrel distortion, image points are displaced in a radial direction fromthe image center.

Different sources of information can be used to obtain cameracalibration. One approach (sometimes called “off-line” calibration) isto use a known target where the three-dimensional world coordinates (orlocations in three-dimensional space) of respective multiple points areknown. One such option may use a checkerboard with known square size ata known location in world coordinates. Such calibration techniquesrequire special equipment and/or a special procedure that is timeconsuming and costly.

Cameras for use in driver assistance and/or driving control may bemounted viewing in the forward direction inside a vehicle behind thewindshield. Stereo calibration for stereo cameras mounted behind thewindshield is thus further complicated; since the windshield distortsthe perspective or camera projection, the calibration may be performedonly after installing the cameras in the host vehicle. Cameras aregenerally modelled using the pinhole camera model using perspectiveprojection. This model is a good approximation to the behavior of mostreal cameras, although in some cases it can be improved by takingnon-linear effects (such as radial distortion) into account.

Auto-calibration or self-calibration refers to a technique in which thecamera parameters are updated “on-line” by processing images beingcaptured during motion of the vehicle. In automotive applications,auto-calibration may insure maintenance-free long-term operation, sincecamera parameters may be subject to drift due mechanical vibrations orlarge temperature variations that are commonly encountered in automotiveapplications. Additionally, reliable auto-calibration techniques mayrender obsolete initial off-line calibration, thus reducing time andcost in the production line.

Thus there is a need for and it would be advantageous to have a methodfor auto-calibration stereo cameras suitable for driver assistance andor driving control applications in automobiles.

Structure-from-Motion (SfM) refers to methods for recoveringthree-dimensional information of a scene that has been projected ontothe back focal plane of a camera. The structural information derivedfrom a SfM algorithm may take the form of a set of projection matrices,one projection matrix per image frame, representing the relationshipbetween a specific two-dimensional point in the image plane of thecamera and its corresponding three-dimensional point in world space.Alternatively, the structure information is the depth or distance to thethree-dimensional (3D) point P=(X,Y,Z) which projects onto the imageplane at the two-dimensional (2D) point p=(x,y). SfM algorithms rely ontracking specific image features from image frame to image frame todetermine structural information concerning the scene.

Structure-from-Motion (SfM) techniques useful in driver assistanceapplications have been previously disclosed by the present Applicant inUS patent application publication 2014/0160244 entitled: Monocular CuedDetection of three-dimensional Structures from Depth Images, which isincluded herein by reference. US patent application publication2014/0160244 discloses a system mountable in a host vehicle including acamera connectible to a processor. Multiple image frames are captured inthe field of view of the camera. In the image frames, an imaged featureis detected of an object in the environment of the vehicle. The imageframes are portioned locally around the imaged feature to produce imagedportions of the image frames including the imaged feature. The imageframes are processed to compute a depth map locally around the detectedimaged feature in the image portions. The depth map may be representedby an image of the feature with a color or grayscale coordinate relatedto a function of distance from the camera to the object. Using thecamera projection and known camera intrinsic and extrinsic parametersrelative to a world coordinate system, the depth map is sufficient toprovide the three-dimensional world coordinates of the imaged feature.

The computation of depth maps from multiple images, either from a motiontime sequence and/or from multiple cameras is the subject of extensiveresearch and numerous systems have been demonstrated.

BRIEF SUMMARY

Various systems and methods are disclosed herein for auto-calibration ofstereo cameras including a first camera and second camera installablebehind the windshield of a host vehicle and oriented to view theenvironment through the windshield, and a processor connectible to thestereo cameras. The processor during motion of the host vehicle capturesmultiple series of images respectively from the stereo cameras includinga first time series of first images from the first camera and a secondtime series of second images from the second camera. The processorlocates multiple first image points of at least one of the first imagescaptured from the first camera at a first time and matches at least aportion of the first image points with first image points of at leastone other of the first images captured from the first camera at a secondtime to produce pairs of corresponding first image points respectivelyin the first images captured at the different times. Responsive to themotion of the host vehicle, the processor computes world coordinatesfrom the pairs of corresponding first image points of at least two ofthe first images captured at the different times from the first camera.The processor matches multiple second image points from the secondcamera to corresponding first image points from the first camera.Matching may be constrained to epipolar lines. The world coordinates ofat least a portion of the first image points are then used to solve formultiple camera parameters of the second camera. The solution of thecamera parameters of the second camera will then predict a depth mapbased on stereo disparity consistent with the world coordinates computedfrom the first images responsive to the motion of the host vehicle. Theprocessor may re-compute the camera parameters of the second camera withan initial estimate of the center of distortion of the second camera tocompute further camera parameters including radial distortion parametersof the second camera. Alternatively, the radial distortion parametersmay be computed together with the camera matrix of the second camera.The processor may perform the auto-calibration while correcting forfurther distortion caused by the first camera and the second camerahaving rolling shutters. The world coordinates may be used asconstraints which correspond to the times when the picture elements ofthe first and second image points are actually captured.

These, additional, and/or other aspects and/or advantages of the presentinvention are set forth in the detailed description which follows;possibly inferable from the detailed description; and/or learnable bypractice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1 illustrates stereo cameras installed behind the windshield of ahost vehicle, according to embodiments of the present invention.

FIG. 2 is a schematic block diagram of a driver assistance and ordriving control system installed in the host vehicle according toembodiments of the present invention.

FIG. 3 illustrates schematically a structure-from-motion (SfM) algorithmas used in embodiments of the present invention.

FIG. 4 illustrates epipolar geometry of the stereo cameras.

FIG. 5 illustrates a simplified flow chart of a method according to anembodiment of the present invention.

The foregoing and/or other aspects will become apparent from thefollowing detailed description when considered in conjunction with theaccompanying drawing figures.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below to explain the presentinvention by referring to the figures.

Before explaining embodiments of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of design and the arrangement of the components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other embodiments or of being practiced or carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein is for the purpose of description and shouldnot be regarded as limiting.

By way of introduction, embodiments of the present invention are basedon the following concept: A pair of stereo cameras is installed behindthe windshield of a host vehicle. The forward motion, distance inmeters, of the host vehicle may be known from sensors in the vehiclethrough the CAN bus. While moving, multiple images from a camera, e.g.one of the two stereo cameras, are used to recover three-dimensionalworld coordinate information of number of world points using Structurefrom Motion (SfM) techniques. SfM thus provides a three-dimensionalcoordinates (X,Y,Z) of world points in the image. This information isthen used to compute the camera parameters of the second camera, and inparticular at least some of the intrinsic and/or extrinsic parameters.Calibration parameters thus determined generate stereo disparity thatgives three-dimensional world coordinate information, e.g. distance ordepth Z that complies with the three-dimensional world coordinateinformation previously computed from SfM.

The resulting solution is a robust depth estimation that merges bothspatial and temporal depth estimation algorithms Following cameracalibration, the depth from stereo disparity may be used to augment thedepth from SfM in particular in situations that are difficult for SfMsuch as when the host vehicle is stationary or when there are multiplemoving objects in the scene.

Referring now to the drawings, reference is now made to FIG. 1 whichillustrates stereo cameras, left camera 12L and right camera 12Rinstalled behind the windshield of vehicle 18 and viewing the roadenvironment substantially in the Z direction in world coordinates, theforward direction of vehicle 18. Reference is now also made to FIG. 2,which is a schematic block diagram of a driver assistance and or drivingcontrol system 16 installed in vehicle 18. A time series of image frames15L is being captured by a processor 14 from the left camera and asecond time series of image frames 15R is being captured by processor14.

Reference is now also made to FIG. 3 which illustrates schematically astructure-from-motion (SfM) algorithm as used in embodiments of thepresent invention. One of the two stereo cameras, in the example stereocamera 12R, is shown in two positions, along a road and installed invehicle 18. A time axis is shown on which camera 12R is shown inposition 0 at time t₀ and later in position 1 at time t₁. An image frame15R₀ is shown schematically which is captured at time t₀ by right camera12R and a second image frame 15R₁ captured at time t_(1.) A worldCartesian coordinate system (X,Y,Z) is shown. At position 1 right camera12R is translated by a vector T and rotated by a matrix R relative toright camera 12R at position 0. A world point P is shown. World point Pis imaged respectively at image point p_(r0) and at image point p_(r1)in both image frame 15R₀ and image frame 15R_(1.)

In the SfM algorithm, multiple, e.g. 6-20 matching image point pairslocated in the two image frames 15R₀ and 15R₁ respectively may berequired. The matching image points p_(r0),p_(r1) are images of multipleobject points in the environment. The matching image point pairp_(r0),p_(r1) as shown as shown in FIG. 3 is just one example.

R is a 3×3 rotation matrix that represents camera orientation and T is athree vector that represents camera translation, of the position of theworld origin O in the right camera coordinate system which istranslated. Together, these are known as camera extrinsic parameters anddescribe camera pose.

Translation vector T and rotation matrix R are suitably parameterizedand the parameters may be determined from the matching image pointsp_(r0),p_(r1) so that with a sufficient number of matching image pointpoint pairs, the parameters of translation vector T and rotation matrixR of right camera 12R may be determined under the assumptions of thepinhole projection The 3D world coordinates (X,Y,Z) for each objectpoint P may be computed from the corresponding points in the two imagesand the translation vector T and rotation matrix R using a variety ofmethods known in the art. Thus, the real world coordinates (X,Y,Z) foreach object point P may be determined from the SfM algorithm using asingle camera 12R. The 3D point P projects to 2D point in the image ofright camera 12R. For simplicity the world coordinate system may bealigned with the right camera resulting in extrinsic camera parameters:M=[I;0] and intrinsic camera parameters may be obtained using nominallens parameters for focal length and lens distortion. Alternatively theintrinsic parameters of the right camera might be obtained using acalibration pattern.

Reference is now made to FIG. 4 which illustrates epipolar geometry ofcameras 12L and 12R. World point P is shown, for instance, the samepoint P as shown in FIG. 3, in which camera 12R is used to determineworld coordinates using an SfM algorithm. In the epipolar geometry asshown, cameras 12L and 12R are modeled using pinhole projection. Thepinhole of left camera 12L is at origin O_(l) which is the origin of theworld coordinate system of left camera 12L. The pinhole of right camera12R is at origin O_(r) which is the origin of the world coordinatesystem of right camera 12R. Image planes 15R and 15L are shown inperspective view. Note that the image plane as shown here in front ofthe optical origins O_(l), O_(r) or pinholes. In a real camera, theimage planes 15R and 15L would be behind the pinholes, and the imageswould be inverted. Image points p_(r) and p_(l) of world object point Pare shown respectively in image planes 15L and 15R respectively. Thefocal lengths (not shown) of cameras 12L and 12R are given by thedistance between the respective origins O_(l), O_(r) to image planes 15Land 15R. The perpendicular to image planes 15L and 15R to the respectiveorigins O_(l), O_(r) defines the principal rays (not shown) and theprincipal points (not shown) in image planes 15L and 15R at the pointsof intersection with the principal rays. The plane formed by originsO_(l), O_(r) and world point P is the epipolar plane with respect toworld point P. The epipolar lines are shown in image planes 15R and 15Lintersecting the epipolar plane O_(l) O_(r)P_(i). The epipoles e_(l) ande_(r) are the points of intersection between epipolar lines and the linebetween the points O_(l), O_(r) of origin.

Reference is now made also to FIG. 5, which illustrates a simplifiedflow chart of a method 50 according to an embodiment of the presentinvention. Instrinsic parameters of right camera 12R are known orassumed for example from camera manufacturer values. During motion ofhost vehicle 18 multiple image frames 15L, 15R are captured (step 51)from respective stereo cameras 12L, 12R mounted in host vehicle 18. Instep 53, multiple image points or multiple image features are located inan image frame 15R. An example of an image point is a corner which maylocated for instance by a Harris operator. The same or similar imagefeatures are located or matched (step 55) in corresponding image pointsin another image frame 15R captured from right stereo camera 12R at asecond time. There are many methods described in the art for matchingimage points and image features. Using an SfM algorithm as known in theart, on images captured from right stereo camera 12R, three-dimensionalworld coordinate information (X,Y,Z) may be computed (step 57) in theworld coordinate system of right stereo camera 12R for the object pointswhich were matched (step 55). In step 59, matching image points orfeatures are located in the left camera images 15L corresponding toimage points or features in right images 15R. Matching points may beobtained simply by comparing an image patch in the neighborhood of thepoint in right image 15R to the best matching patch in left image 15L.Projective epipolar constraints may be used in step 59, although thecalibrated epipolar geometry has not yet been solved. The worldcoordinate information (X,Y,Z) of image points as computed (step 57)using SfM in right images 12R are projected onto the left image 12L, andcompared to the matching points in the left image frame and the distanceminimized to solve (step 60) for the camera parameters of left camera12L. Step 60 may be performed while varying camera parameters of leftcamera 12L and when the best solution is found, camera parameters 12L ofthe left camera are output. Alternatively, a closed form solution (step60) is determined for the camera projection matrix of the left cameraand other camera parameters of the left camera using the 3D points andtheir 2D image points in the left camera. The left camera parameters canbe used to describe the epipolar geometry. A stereo disparity mapcomputed from left and right stereo cameras 12L, 12R results in a depthmap consistent with the three-dimensional world coordinate information,e.g. depth Z_(r) computed using SfM with right camera 12R.

Further Embodiments and Features

Matching (steps 55 and 59) has been described as being between sparselydistributed image features and/or points in consecutive images 15R orleft 15L/right 15R image pairs. However, image points may be selectedwhich are densely distributed and dense optical flow techniques may beused for the SfM computation (step 57) and/or for computations usingstereo pairs with the same general structure. Dense optical flow may becomputed between two right images 15R a dense depth map may be computed.Dense optical flow may similarly be computed using respective pairedimages from stereo cameras 12L, 12R.

The use of right 12R and left 12L stereo cameras is by way of example.In different embodiments the present invention is applicable to stereocameras relatively displaced forward/rearward and/or displacedvertically such as a stereo pair of a windshield camera and camerainstalled on a car bumper.

Although the discussion includes modelling of radial distortion, themethods as disclosed herein may be similarly applied to other lensdistortion models such as a fish eye lens. However, a closed formsolution may not be available for all types of distortion and numericalsolution is available.

Consider a windshield and bumper camera pair in which the bumper camerais a wide fish eye camera. It is possible that the objects at the edgeof the bumper camera image are not visible in the windshield cameraimage. However these objects may have been visible in earlier images ofthe windshield camera when the vehicle was farther from the object. Ifthe three-dimensional position of the object was detected in earlierimages of the windshield camera and the objects are matched in earlierimages, the objects can be tracked and the three-dimensional positionmay be updated using ego motion from an SfM computation from images ofthe bumper camera. The updated three-dimensional positions may then beused at a later image for the calibration of the windshield camera.

A More Formal Description

Method 50 according to embodiments of the present invention is presentedmore formally and in further detail in the description as follows. Rulesof notation in the description as follows are :

Square brackets [ . . . ] are used to denote a matrix. A vector isdenoted as a matrix of a single column. Comma may be used to separatecolumns of a matrix. Semi-colon may be used to separate rows of amatrix. The symbol

refers to real space. Points in real space

are represented by vectors or matrices of single column. Coordinates inEuclidean or three dimensional (3D) world space are denoted with capitalletters, e.g. X,Y,Z. Coordinates in two-dimensional (2D) image space aredenoted with small letters, e.g. x,y.

In what follows, it will be convenient to work with homogeneous as wellas Euclidean coordinates. In homogeneous coordinates, a point inN-dimensional space is expressed by a vector with N+1 elements that isdefined only up to scale, i.e. multiplying the vector by an arbitrarynon-zero scale factor will not change its meaning. Provided the N+1'thelement is non-zero, a homogeneous coordinate may be related to itsEuclidean equivalent by dividing the first N elements by the N+1'th.Otherwise, the coordinate describes a point at infinity.

Calibration Parameters

A 3D world point P=[X;Y;Z] is mapped to a 2D image point p=[x; y] by a3×4 projection matrix M.[p; 1]≅M[P; 1]  (1)where [p;1] is a 3-vector in homogeneous coordinates in image space,[P;1] is a 4-vector in world space in homogeneous coordinates and ≅denotes equality up to scale. The matrix M may be further uniquelydecomposed into: an intrinsic parameters matrix A∈

^(3×3), a rotation matrix R∈

^(3×3) and a translation vector t∈

^(3×1) as follows:M=A[R,t]  (2)

The intrinsic parameters matrix A is a general upper triangular matrix.Given focal length f and principal point [pp_(x), pp_(y)], the intrinsiccamera parameters matrix A becomes:

$\begin{matrix}{A = \begin{bmatrix}f & 0 & {pp}_{x} \\0 & f & {pp}_{y} \\0 & 0 & 1\end{bmatrix}} & (3)\end{matrix}$

The distortion of the camera is modeled by a radial distortion. Thedistortion parameters θ={c_(x), c_(y), α₀, α₁} are modeled by a centerof distortion point {tilde over (c)}=[{tilde over (c)}_(x);{tilde over(c)}_(y)] and two stretch coefficients α₀ and α₁. A function F tocorrect a distortion of a point {tilde over (p)} is given by:p=F({tilde over (p)}; θ)   (4)F({tilde over (p)}; θ)=({tilde over (p)}−{tilde over (c)})(1+α₀ r ²+α₁ r⁴)+{tilde over (c)}  (5)wherer=∥{tilde over (p)}−{tilde over (c)}∥ ₂which is the least squares or L² norm of the difference between point{tilde over (p)} and center point {tilde over (c)}. Taking distortioninto account, equation 1becomes:λ[F({tilde over (p)}; θ);1)]=M[P;1]  (6)where λ is the missing scale factor from equation 1.

For simplicity of notation the above equation may be written in thefollowing manner:F({tilde over (p)}, θ)=dehom(M·hom(P))   (7)hom(p)=[p; 1]dehom(λ[p; 1])=pwhere the function horn denotes a conversion from Euclidean coordinatesto homogeneous coordinates and the function dehom denotes a conversionfrom homogeneous coordinates back to Euclidean coordinates.

The Algorithm

In automotive applications, the baseline is relatively small and theobjects relatively distant. As a result image disparity is quite small.As a result, feature points in the right image can be matched to pointsin the left image using image tracking such as the Lucas-Kanade methodor exhaustive search for a similar patch between the two images in asmall search region. In the case of larger potential disparities,scale-invariant feature transform (SIFT) features may be used to assistin the matching. Many other methods for point matching are known in theart.

Let {tilde over (p)}_(r), {tilde over (p)}_(r)∈

denote a pair of matching points in the right and left images, capturedby the right and left stereo cameras, respectively. Let Z_(r)∈

denote the depth value related to the matching point {tilde over(p)}_(r) as computed by a mono or single camera structure-from-motion(SfM) algorithm. The intrinsic parameter matrices of the right cameraA_(r), as well as the distortion parameters θ_(r) and θ_(l) of right andleft cameras respectively are initialized to the default parameters ofthe camera manufacturer.

Camera Matrix Initialization

From the known motion of the cameras, depth Z_(r) is computed using thestructure-from-motion (SfM) algorithm. Thus, the world coordinate systemis defined to be the coordinate system of the right camera with norotation nor translation:

$\begin{matrix}{M_{r} = {A_{r}\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix}}} & (8)\end{matrix}$

Intrinsic parameters matrix A_(r) and distortion parameters θ_(r) of theright camera are used to compute the world point P from the distortedimage point {tilde over (p)}_(r) and the depth Z_(r), which is computedby the SFM algorithm. Specifically, the vector direction V_(r) of thedistorted image point {tilde over (p)}_(r) is given by:V _(r) =A _(r) ⁻¹hom(F({tilde over (p)} _(r), θ_(r)))   (9)

And given the depth Z_(r) we have the vector of the world point P_(r)corresponding to the distorted image point {tilde over (p)}_(r) in theimage of the right camera:

$\begin{matrix}{P_{r} = {\frac{Z_{r}}{V_{rz}}V_{r}}} & (10)\end{matrix}$

Now, we wish to recover the left camera projection matrix M_(l) giventhe 3D points P_(r), in the world coordinate system, and theircorresponding undistorted image points, [x,y]=F({tilde over(p)}_(l),θ_(l)) in the coordinate system of the left camera. Theequations take the following form:

Let M_(l)=[m_(l0) ^(T);m_(l1) ^(T);m_(l2) ^(T)] where m_(l1) denotes thei-th row of matrix M_(l).

We have x=(m_(l0) ^(T)P)/(m_(l2) ^(T)P) and y=(m_(l1) ^(T)P)/(m_(l2)^(T)P)

From which we obtain:m _(l0) ^(T) P−x·m _(l2) ^(T) P=0   (11)m _(l1) ^(T) P−y·m _(l2) ^(T) P=0   (12)

We use least square solution to these equations to solve the left cameramatrix: M_(l). We combat outliers by using RANSAC. In each round wesample 6 points, and solve the camera matrix M_(l) using Equations (11)and (12). The score for a solution in each round is computed as follows:The distance of the warped projected world point from the matched imagepoint is:dist_(i) =∥{tilde over (p)} _(li) −F ⁻¹(dehom(M _(l)·hom(P _(i))),θ_(l)|₂   (13)

A simple score that we can define for each model is:

$\begin{matrix}{{score} = {\sum\limits_{i}{\min\left( {{dist}_{i},\tau} \right)}^{2}}} & (14)\end{matrix}$where κ is a specified threshold on the error of inlier points.Alternatively, we use the following score:

$\begin{matrix}{{score} = {\sum\limits_{i}\sqrt{\tau^{2} - {\max\left( {{\tau - {dist}_{i}},0} \right)}^{2}}}} & (15)\end{matrix}$

The score defined in Equation (15) has the advantage that inliers thathave high distance, which are more likely to be outliers, have lessimpact on the score, thus the final parameters are likely to be moreaccurate.

Camera Matrix and Distortion Correction Initialization

We compute the distance error specified by Equation (13), and using apredefined threshold we determine which points pair are inliers.

Given the inliers we re-compute the parameters of the left camera,without using the computed camera matrix M as an initial guess. The onlyinput that we use is an initial guess to distortion center. Thealgorithm in this section is divided into two steps. In the first stepwe have a non linear estimation of the distortion center. In the secondstep we efficiently estimate the rest of the calibration parameters.

The camera matrix {circumflex over (M)} that is recovered by these stepsis relative to a given distortion center: [c_(x); c_(y)]. That meansthat projection model is:p _(l) −[c _(x) ;c _(y)]=dehom({circumflex over (M)}·hom(P))   (16)

Once we reveal the camera matrix we can modify it to be relative to theimage:

$\begin{matrix}{M = {\begin{bmatrix}1 & 0 & c_{x} \\0 & 1 & c_{y} \\0 & 0 & 1\end{bmatrix}\hat{M}}} & (17)\end{matrix}$

We now describe how we compute an error score for the non linear search,given the distortion center. Let [c_(x); c_(y)] be the distortioncenter. Let[{tilde over (x)}; {tilde over (y)}]={tilde over (p)} _(l) −[c _(x) ; c_(y)]be the distorted point in the left image, relative to the distortioncenter. We seek for camera matrix {circumflex over (M)} which isrelative to the distortion center thus the vector[x;y]=dehom({circumflex over (M)}·hom(P))is the projected point relative to the distortion center, and as aresult it is a scale of the vector[{tilde over (x)}; {tilde over (y)}]

Thus we can define a linear equation over two rows of the camera matrix:{circumflex over (M)}=[m₀ ^(T);m₁ ^(T);m₂ ^(T)]:{circumflex over (M)}·hom(P)=λ·hom([{tilde over (x)}; {tilde over (y)}])(m ₀ ^(T)·hom(P))/(m ₁ ^(T)·hom(P))={tilde over (x)}/{tilde over (y)}

And we have the linear equation:m ₀ ^(T)·hom(P)·{tilde over (y)}−m ₁ ^(T)·hom(P)·{tilde over (x)}=0  (18)After we solve m₀ and m₁, we can define a simple error function bycomputing the distance of the point ({tilde over (p)}_(i)−c) from theray defined by v=[m₀ ^(T);m₁ ^(T)]·hom(P):

$\begin{matrix}{{dist} = {{\left( {I - \frac{\upsilon \cdot \upsilon^{T}}{{\upsilon }^{2}}} \right)\left( {{\hat{p}}_{l} - c} \right)}}} & (19)\end{matrix}$

The error that is defined by Equation (15), is minimized by non linearoptimization to obtain the distortion center [c_(x); c_(y)].

Complete Parameters Estimation

The input to this step are the inliers points: P and {tilde over(p)}_(l) the center of distortion [c_(x); c_(y)], and two rows of thecamera matrix: m₀ and m₁. The output of this step is the third row ofthe camera matrix m₂, and the distortion stretch coefficients: α₀ andα_(1.) See equation (5).

Recall that {circumflex over (M)} is relative to the distortion center,thus from equations (5) and (7) we have that:[{tilde over (x)};{tilde over (y)}](1+α₀ r ²+α₁ r ⁴)=dehom({circumflexover (M)}·hom(P))   (20)so we get two equations for each pair of points:{tilde over (x)}(1+α₀ r ²+α₁ r ⁴)(m ₂ ^(T) P)=m ₀ ^(T) P{tilde over (y)}(1+α₀ r ²+α₁ r ⁴)(m ₂ ^(T) P)=m ₁ ^(T) P

Where we have 6 unknowns: The 4 unknowns of m₂, and the 2 unknowns α₀and α₁. These equations are not linear and instead we solve linearly forthe following 12 unknowns:[m₂ ^(T); α₀m₂ ^(T); α₁m2^(T)]

$\begin{matrix}{\left\lbrack {U,D,V} \right\rbrack = {{svd}\left( \begin{bmatrix}b_{0} & b_{1} & b_{2} & b_{3} \\b_{4} & b_{5} & b_{6} & b_{7} \\b_{8} & b_{9} & b_{10} & b_{11}\end{bmatrix} \right)}} & (21)\end{matrix}$

Once we have the solution vector: b∈

² we decompose it using Singular Value Decomposition (SVD):

The solution to the 6 parameters: m₂, and α₀ and α₁, can be taken usingthe highest singular value and its related singular vectors:U₀D₀V₀ ^(T)≅[1;α₀;α₁]m₂ ^(T)   (22)

4.4 Refine Camera Parameters

We refine the parameters of the left camera using non linearoptimization. The optimizer minimizes the same score define in Equation(15).

Note that we optimize only the left camera parameters, and use the rightcamera parameters as they were given in the initialization step. Thereis also the option to refine the right camera parameters, but thisoption is more expensive to compute.

Correction for Distortion Caused by Rolling Shutter Cameras

The cameras that we use have a rolling shutter effect. That means thatrows of camera are not captured at the same time, but captured isperformed row after row. In order to perform auto calibration for thistype of camera we fix the input to the algorithm in the followingmanner. The Structure From Motion (SfM) algorithm computes the depth ofeach pixel. In equation (10) we called this depth Z_(r). The depthreported by SfM is a depth at specific time related to the image, suchthat the depths of all pixels are generated for an imaginary imagewithout the effect of rolling shutter. We do not use this depth. Insteadwe use a depth that is also computed by SfM algorithm, which is thedepth at the time when the pixel was actually captured. Let us call thisdepth {circumflex over (Z)}_(r). Otherwise, the equation as follows hasthe same form as equation (10):

$\begin{matrix}{{\hat{P}}_{r} = {\frac{{\hat{Z}}_{r}}{V_{rz}}V_{r}}} & (23)\end{matrix}$

The world point {circumflex over (P)}_(r) is related to time when thepixel was actually captured. This world point is still not the correctworld point to use. The problem is that due the rolling shutter, and theinability of generating perfectly aligned stereo cameras, the stereocamera are actually not perfectly synchronized, and thus the world point{circumflex over (P)}_(r) is not the point that is projected to p_(l).In order to fix it we use another output from SfM, the ego motion matrixT=[R, t] that maps point from previous image to current image:P _(c) =R·P _(p) ÷t   (24)

Where P_(c) and P_(p) are the coordinates of a stationary point relativeto the current and previous coordinate systems respectively.

Let Δ_(SFM)=t_(c)−t_(p) be the time elapsed between previous image andcurrent image used by the SfM. t_(c) and t_(p) are the times related tocurrent and previous images respectively, in seconds.

Let Δ_(stereo)=t_(l)−t_(r) be the time discrepancy between left andright stereo images, due to non perfect synchronization. t_(l) and t_(r)are the times related to left and right stereo images respectively, inseconds.

We can fix the world point

$\begin{matrix}{P_{r} = {{dehom}\left( {T^{\frac{\Delta_{stereo}}{\Delta_{SFM}}} \cdot {\hom\left( {\hat{P}}_{r} \right)}} \right)}} & (25)\end{matrix}$that is related to time when the pixel in the right image was captured,to be a world point P_(r) that is related to the time when the matchingpixel in the left image was captured as follows:

Where

$T = \begin{bmatrix}R & t \\{0\mspace{14mu} 0\mspace{14mu} 0} & 1\end{bmatrix}$

The world point P_(r) defined by equation (25), replaces the world pointP_(r) defined by equation (10). The rest of the auto calibrationalgorithm remains unchanged.

The term “object” as used herein refers to an object in real space beingviewed by a camera. A curb along the edge of a road and a lane marker inthe road are examples of objects. The term “image” refers to the imageof one or more objects in image space at the focal plane of camera 12.Image coordinates (x,y) in small letters refer to image space and may bein arbitrary units or numbers of picture elements in the horizontal andvertical directions with the pixel dimensions assumed. The term “imagemotion” refers to motion of an image of an object in image space. Fromimage frame 15 to a subsequent image frame 15 the points of the image ofthe object may map from one set of coordinates (x1,y1) to a differentset of coordinates (x2,y2). The term “image motion” refers to themapping of coordinates of an image from image frame to image frame or afunction of the mapping.

The term “projection” or “projecting” as used herein refers to camera orperspective projection from three dimensional space to a two dimensionalimage unless otherwise indicated by the context.

A “depth map” as used herein is an image that contains informationrelating to the world space distance of object points of scene objectsfrom a viewpoint.

Image points in the stereo images of the same world object point orfeature are known herein as as “corresponding” or “matching” points.

The indefinite articles “a” and “an” is used herein, such as “a camera”,“an image frame” have the meaning of “one or more” that is “one or morecameras” or “one or more image frames”.

Although selected embodiments of the present invention have been shownand described, it is to be understood the present invention is notlimited to the described embodiments. Instead, it is to be appreciatedthat changes may be made to these embodiments, the scope of which isdefined by the claims and the equivalents thereof.

What is claimed is:
 1. A method for auto-calibration of stereo camerasincluding a first camera and second camera installable behind thewindshield of a host vehicle and orientable to view the environmentthrough the windshield, the method comprising: during motion of the hostvehicle, capturing by a processor multiple series of images respectivelyfrom the stereo cameras including a first time series of first imagesfrom the first camera and a second time series of second images from thesecond camera; locating a plurality of first image points of at leastone of the first images captured from the first camera at a first time;matching at least a portion of the first image points with first imagepoints of at least one other of the first images captured from the firstcamera at a second time to produce pairs of corresponding first imagepoints respectively in the first images captured at the different times;responsive to the motion of the host vehicle, computing worldcoordinates from the pairs of corresponding first image points of atleast two of the first images captured at the different times from thefirst camera; matching a plurality of second image points in at leastone of the second images captured from the second camera so that thematching second image points correspond to at least a portion of thefirst image points, using the world coordinates determined from thefirst camera solving for a plurality of camera parameters of the secondcamera.
 2. The method of claim 1, wherein the solution of the cameraparameters of the second camera predicts a depth map based on stereodisparity consistent with the world coordinates computed from the firstimages responsive to the motion of the host vehicle.
 3. The method ofclaim 2, further comprising: re-computing the camera parameters of thesecond camera with an initial estimate of the center of distortion ofthe second camera to compute further camera parameters including radialdistortion parameters of the second camera.
 4. The method of claim 2,wherein said solving the camera parameters includes solving distortionparameters intrinsic to the second camera.
 5. The method of claim 2,further comprising: performing the auto-calibration while correcting forfurther distortion caused by the first camera and the second camerahaving rolling shutters.
 6. The method of claim 5, further comprising:using the world coordinates as the constraints which correspond to thetimes when the picture elements of the first and second image points areactually captured.
 7. A system for auto-calibration of stereo camerasincluding a first camera and second camera installed behind thewindshield of a host vehicle and oriented to view the environmentthrough the windshield, and a processor connectible to the stereocameras and operable to: during motion of the host vehicle, capturemultiple series of images respectively from the stereo cameras includinga first time series of first images from the first camera and a secondtime series of second images from the second camera; locate a pluralityof first image points of at least one of the first images captured fromthe first camera at a first time; match at least a portion of the firstimage points with first image points of at least one other of the firstimages captured from the first camera at a second time to produce pairsof corresponding first image points respectively in the first imagescaptured at the different times; responsive to the motion of the hostvehicle, compute world coordinates from the pairs of corresponding firstimage points of at least two of the first images captured at thedifferent times from the first camera; match a plurality of second imagepoints in at least one of the second images captured from the secondcamera so that the matching second image points correspond to at least aportion of the first image points using the world coordinates determinedfrom the first camera, to solve for a plurality of camera parameters ofthe second camera.
 8. The system of claim 7, wherein the solution of thecamera parameters of the second camera predicts a depth map based onstereo disparity consistent with the world coordinates computed from thefirst images responsive to the motion of the host vehicle.
 9. The systemof claim 8, wherein the processor is further operable to re-compute thecamera parameters of the second camera with an initial estimate of thecenter of distortion of the second camera to compute further cameraparameters including radial distortion parameters of the second camera.10. The system of claim 8, wherein the camera parameters of the secondcamera include intrinsic distortion parameters.
 11. The system of claim8, wherein the processor is further operable to perform theauto-calibration while correcting for further distortion caused by thefirst camera and the second camera having rolling shutters.
 12. Thesystem of claim 11, wherein the world coordinates are used asconstraints which correspond to the times when the picture elements ofthe first and second image points are actually captured.